A framework for analyzing embodied communicative feedback in multimodal corpora

Communicative feedback refers to unobtrusive (usually short) vocal or bodily expressions whereby a recipient of information can inform a contributor of information about whether he/she is able and willing to communicate, perceive the information, and understand the information. This paper provides a theory for embodied communicative feedback, describing the different dimensions and features involved. It also provides a corpus analysis part, describing a first data coding and analysis method geared to find the features postulated by the theory. Author keywords Communicative embodied feedback, contact, perception, understanding, emotions, multimodal, embodied communication INTRODUCTION The purpose of this paper is to present a theoretical model of communicative feedback, which is to be used in a VR agent capable of multimodal communication. Another purpose is to briefly present the coding categories which are being used to obtain data guiding the agent’s behavior. Below, we first present the theory. The function/purpose of communication is to share information. This usually takes place by two or more communicators taking turns in contributing new information. In order to be successful, this process requires a feedback system to make sure the contributed information is really shared. Using the cybernetic notion of feedback of Wiener (1948) as a point of departure, we may define a notion of communicative feedback in terms of four functions that directly arise from basic requirements of human communication: Communicative feedback refers to unobtrusive (usually short) vocal or bodily expressions whereby a recipient of information can inform a contributor of information about whether he/she is able and willing to (i) communicate (have contact), (ii) perceive the information (perception), and (i i i ) understand the information (understanding). In addition, (iv) feedback information can be given about emotions and attitudes triggered by the information, a special case here being an evaluation of the main evocative function of the current and most recent contributions (cf. Allwood, Nivre & Ahlsén 1992 and Allwood 2000, where the theory is described more in detail). The central role of feedback in communication is underpinned already by the fact that simple feedback words like yes, no and m are among the most frequent in spoken language. A proper analysis of their semantic/pragmatic content, however, is fairly complex and involves several different dimensions. One striking feature is that these words involve a high degree of context dependence with regard to the features of the preceding communicative act, notably the type of speech act (mood), its factual polarity, information status and evocative function (cf. Allwood, Nivre & Ahlsén 1992). Moreover, when studying natural face-to-face interaction it becomes apparent that the human feedback system comprises much more than words. Interlocutors incessantly coordinate and exchange feedback information by nonverbal means like posture, facial expression or prosody. In this paper, we extend the theoretical account developed earlier to cover embodied communicative feedback and provide a framework for analyzing it in multimodal corpora. DIMENSIONS OF COMMUNICATIVE FEEDBACK Communicative feedback can be characterized with respect to several different dimensions. Some of the most relevant in this context are the following: